To assure the quality and accuracy of computer software products and of systems utilizing such products, it is necessary that such products and systems be thoroughly tested before being delivered to customers. However, as such software products become more complex and in particular where these software products are being utilized in a highly interactive and integrated system, the testing of such products becomes increasingly complex and time-consuming. Automated software testing programs and procedures are, therefore, becoming essential in quality assurance work.
However, particularly in integrated software systems, it is important that the testing be performed in the same manner and under the same conditions as the system will be used by the customer. Thus, the various program products cannot be tested discretely, but must be tested as part of the integrated system in a typical operating environment.
Another problem is that the most probable locations for error are the points of interface of the software product with the other software products and with the user. It is, therefore, desirable to be able to run a software system in a test mode which permits looking at all inputs to the system as they are generated. While some test technologies have been developed which allow the capture and playback of program interactions, these technologies have not functioned well at capturing mouse movement and mouse and menu "point and click" paradigm. They also do not have the flexibility to permit production code as opposed to test code to be run in the same manner as such programs would be run by the customer and, in particular, do not address an integrated tool environment. Finally, such techniques generate huge log files and lack flexibility to permit a person doing testing to move forward and backward in the software products and in the procedures as desired to isolate problems, or to add new inputs at various points in the procedure to verify suspected problems, to assist in isolating problems or for other purposes.
Another technique sometimes used for testing software involves generating command scripts which are a sequence of commands put together in a format which the software product or tool understands. Such command scripts are designed to work with only a single software tool and do not provide either for interactions with the user or with other tools in an integrated system. Further, once a command script is started, the user loses control until the script is ended since the command scripts do not return control to the user.
A need, therefore, exists for an improved technique for testing complex integrated software systems involving a number of interactive program products or tools, which technique permits the system to be operated in any desired environment; and which permits mouse and other inputs to be collected for replay in a test environment. However, the system should at all times provide the user with the capability to stop the operation and make any entries which the user/tester wishes in either record or playback, to move forward or backward in a playback operation, to skip over selected portions of the operation, to run the playback in a step-by-step mode with steps of selected size, and to provide the user with other capabilities which permit him to retain complete control of the testing operation. Such a system should be flexible enough to operate with any, or substantially any, software products so that new software tools do not have to be generated each time a new software product is added to the system, or some other change is made in the system. The system should permit all of the software products in the system to be tested together or the testing of any selected combination of the software products available in the system. Further, in addition to being useful for testing the system, such a flexible playback capability would also be useful in performing demonstrations of a system or for training personnel in the use of a system.